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ABSTRACT 


The Officer Strategic Analysis Model (OSAM) is a powerful entity-based, 
time-stepped simulation model that takes into account actual officer inventory 
and allows the application of several types of annual personnel plans and force¬ 
shaping policy to project officer end strength and composition. This thesis 
determines whether OSAM can be modified from its current deterministic version 
to be a valuable tool for the Navy’s Strategic Resourcing Branch (N100) officer 
strength planning. This research utilizes experimental design and data farming 
techniques to “grow” big data via a stochastic modification of OSAM and applies 
proven methods of statistical analysis to explore the interactions between natural 
losses, accession methods, and forced losses. Using analysis of variance and 
regression tree modeling, the standard deviation from the mean end strength is 
found to be heavily dependent upon accession method and simulation year. 
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EXECUTIVE SUMMARY 


Navy officer manpower and personnel planning is an extremely complex 
process with high costs for error. The Office of the Chief of Naval Operations’ 
(OPNAV) Strategic Resourcing Branch (N100) is responsible for generating 
inventory projections and keeping them up-to-date throughout the fiscal year 
(FY). This forecast is the basis for the manpower budget requested yearly in the 
Navy’s Program Objective Memorandum (POM) and ultimately for the annual 
end strength authorized by Congress. This authorized end strength is deemed to 
be the minimum necessary to fulfill the Navy’s responsibilities as set forth in the 
National Defense Strategy and is regulated by law. Once broken down by 
designator and pay grade, the officer end-strength authorization is referred to as 
Officer Programmed Authorizations (OPA). Any deviation below OPA is 
considered to be a capabilities risk while deviation above creates a fiscal risk to 
the account used to fund personnel, the so-called military personnel Navy (MPN) 
account. 

In order to reduce risk, N100 has several manpower, personnel, training, 
and education (MPTE) modeling resources available. One such model that is 
currently under-utilized as a deterministic model is the Officer Strategic Analysis 
Model (OSAM). OSAM is a powerful entity-based, time-stepped simulation model 
that takes into account actual officer inventory and allows the application of 
several types of annual personnel plans and force-shaping policy to project 
anticipated end strength and composition within or against OPA constraints. 
Unfortunately, it has been programmed to reduce the randomness inherent in the 
model, thus resulting in a single array of values for every set of input parameters. 
The user loses valuable insight that could be gained by studying the variation of 
outputs provided by a stochastic model, or a model that implements probability 
distributions to determine outcomes. 

This research is conducted utilizing a modified version of OSAM to allow 
for stochastic variation. The study employs the resources of the Simulation 
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Experiments & Efficient Designs (SEED) Center for Data Farming 
( http.V/harvest.nps.edu) at the Naval Postgraduate School (NPS) to data farm, or 
“grow,” vast amounts of data utilizing design of experiments and cluster 
computing. Graphical and statistical analysis completed utilizing JMP 12.1.0 is 
used to infer relationships between the factors and their contribution to the 
variation of output. Finally, the standard deviation of the mean end strength is 
modeled by a regression tree and tested for predictive capability. 

Surprisingly, we find that variation of loss rates has little to no effect on end 
strength unless manipulated in conjunction with an accession plan. The large 
flexibility of the other accession methods can greatly compensate for any effect 
created by the loss rate alone. Because of the unusual responses created by the 
constrained and unconstrained accessions methods. Probationary Officer 
Continuation and Re-designation (POOR) board losses and the proposed 
enhanced POOR (EPOCR) policy could only be adequately observed in 
conjunction with the designated accession plan. Even with the most aggressive 
EPOCR policy losses, officer over-execution would remain constant at predicted 
loss rates or continue to increase if loss rates are lower than predicted. If loss 
rates are greater than predicted, additional EPOCR losses could cause an end 
strength significantly below OPA. Should this policy be implemented, natural loss 
rates will have to be carefully monitored to avoid over-correction of end strength. 

It is evident throughout every manipulation of the results that the 
constrained accessions method is the accession option that most easily meets 
OPA. Modeling with this parameter shows an immediate return to OPA and 
eliminates the need for force-shaping policy. Flowever, an in-depth study into the 
accession values created by this option and their future repercussions is required 
before any type of implementation. 
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I. INTRODUCTION 


Navy officer manpower and personnel planning is an extremely complex 
process with high costs for error. The Office of the Chief of Naval Operations’ 
(OPNAV) Strategic Resourcing Branch (N100) is responsible for generating 
inventory projections and keeping them up-to-date throughout the fiscal year 
(FY). This forecast is the basis for the manpower budget requested yearly in the 
Navy’s Program Objective Memorandum (POM) and ultimately for the annual 
end strength authorized by Congress. This authorized end strength is deemed to 
be the minimum necessary to fulfill the Navy’s responsibilities as set forth in the 
National Defense Strategy. In addition, by law, actual end strength must fall 
within 3% above or 0.5% below this authorization (Title 10 United States Code). 
Once broken down by designator and pay grade, the officer end-strength 
authorization is referred to as Officer Programmed Authorizations (OPA). Any 
deviation below OPA is considered to be a capabilities risk while deviation above 
creates a fiscal risk to the account used to fund personnel, the so-called military 
personnel Navy (MPN) account. 

The Navy is currently operating with fiscal risk to the MPN account caused 
by officer end-strength inventory exceeding programmed end-strength. This is 
also known as officer over-execution (OOE) (D. Clark, personal communication, 
April 17, 2015). OOE occurs for a variety of reasons, the most obvious being 
inadequate loss projections. OOE has been a concern for over a decade (O’Neal, 
2003). Current officer distributions show over-execution in Surface, Undersea, 
and Air communities. The over-execution of aviation officers is mostly in control 
grade officer ranks (04-06) and therefore easier to correct, whereas the 
submariner and surface officers are over-executing in the junior pay grades. Due 
to the sheer size of their community, junior grade surface warfare officers 
(SWOs) constitute the majority of the OOE problem. 

While over-execution appears to be the historical norm for officer inventory 

absent any sort of force-shaping measure, OOE has recently garnered more 
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attention (P. Saluke, personal communication, May 28, 2015). Previously, OOE 
has been shielded by several budgetary factors. The past decade of continuous 
operations provided a large Overseas Contingency Operations (OCO) budget, 
which has been reduced as a result of the military drawdown. Additionally, 
the Navy has been under-accessing certain designators, particularly in the 
restricted line, and was comprised of a more junior force than budgeted, both of 
which contributed to a fiscal cushion that offset OOE (P. Saluke, personal 
communication. May 28, 2015). 

In addition to tightening fiscal constraints, the evolving structure of the 
SWO community is exacerbating OOE. Low retention rates already require three 
SWO ensigns to every department head billet, meaning the Navy must over¬ 
access in order to meet the SWO control grade officer demand (Moran, 2014). 
The elimination of the brick-and-mortar SWO basic school, combined with the 
more senior officer manning requirements of the Littoral Combat Ship, removes 
many of the billets once programmed to develop junior SWOs. This is creating a 
manning “diamond,” where the most manning is required in the control grade 
billets, rather than the traditional pyramid, where natural losses groomed a large 
junior force into the required control grades. This new paradigm leaves fewer and 
fewer programmed junior grade billets to develop senior officers. 

A. PROBLEM STATEMENT 

The current economic environment generates concern that OOE cannot 
continue unchecked indefinitely. Due to fear of outside intervention by the Office 
of the Secretary of Defense and Congress, POM16 requests additional end 
strength to close the gap between authorized and projected officer inventory. 
Even with this investment, OOE costs are projected to exceed the budget by 
nearly $2 billion from FY15 to FY19, still an undesirable outcome (P. Saluke, 
personal communication. May 28, 2015). 

N100 has been tasked with investigating force-shaping policy to correct 
OOE over the period of seven years from FY15 to FY 21. Several internal studies 
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have already been completed with some of the most viable courses of action 
dealing with the Probationary Officer Continuation and Re-designation (POOR) 
boards. Currently, around 400 junior officers are referred to POCR boards each 
FY. Communities that rely on lateral transfers for manning require approximately 
160 of those officers to be re-designated while the remaining 240 officers 
are separated. There is a proposal to create an enhanced POCR (EPOCR) 
board, which would increase the number of SWOs referred with 4-6 years of 
commissioned service (YCS) by either 60 or 120, thereby increasing the number 
of separations. Based on the studies’ results, 120 additional separations would 
help inventory converge toward OPA, although there would still be a slight OOE 
(P. Saluke, personal communication. May 28, 2015). 

These initial projections were produced with basic spreadsheet modeling. 
However, N100 has several manpower, personnel, training, and education 
(MPTE) modeling resources available. One such model that is currently under¬ 
utilized as a deterministic model is the Officer Strategic Analysis Model (OSAM). 
OSAM is a powerful entity-based, time-stepped simulation model that takes into 
account actual officer inventory and allows the application of several types of 
annual personnel plans and any force-shaping policy to project anticipated end 
strength and composition within or against OPA constraints. Unfortunately, it has 
been programmed to reduce the randomness inherent in the model, thus 
resulting in a single array of output values for every set of input parameters. The 
user loses valuable insight that could be gained by studying the variation of 
outputs provided by a stochastic model, or a model that implements probability 
distributions to determine outcomes. In its current form, OSAM is primarily used 
by the Strategic Assessment Group (SAG) to predict the effects of manpower 
changes on small communities within the officer corps, but this sophisticated 
simulation could provide a complete analysis of OOE force shaping policy effects 
to N100 as well. 
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B. THESIS PURPOSE 


This thesis determines whether OSAM can be modified from the version 
utilized by SAG to be a valuable tool for N100. Currently, its models lack the 
ability to simulate aggregate officer inventory projections that provide second- 
and third-order effects only available through a complex entity-based simulation 
like OSAM. OSAM already utilizes a powerful multi-parameter simulation to 
project inventory, but lacks any sort of measure of accuracy with which to qualify 
the resulting data. Additionally, once the actual operating environment differs 
from that of the simulated factors, by a higher or lower natural loss rate for 
example, the end of year inventory (EOY) projections are no longer valid. 
Conducting data analysis over a large amount of stochastically farmed data will 
provide insight into the variability of projections. Specifically, a design of 
experiments is constructed to explore variations over natural loss rates and yield 
quantitative ranges of accuracy for OSAM inventory forecasts. In particular to the 
OOE problem, interpreting the interactions between natural losses, accession 
methods, and forced losses via metamodeling will not only provide data with 
which to judge the proposed EPOCR force-shaping policy but will also offer 
context within which alternative policies can be considered. 

C. RESEARCH QUESTIONS 

The inventory of Navy officers is in a constant state of flux, and yet 
strength planners must use imprecise methods of forecasting to develop the 
quality and quantity of officers needed in tomorrow’s Navy. This research 
attempts to quantify the accuracy of these predictions and access specific force¬ 
shaping policies by exploring the following questions: 

(1) Can OSAM be run stochastically to provide a measure of accuracy 
for multi-year summary (MYS) values? 

(2) What impact does variation of loss rates (as expressed in loss 
plans) have on end of year inventory? 
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(3) What impact do Probationary Officer Continuation and Re¬ 
designation Board losses have on end of year inventory? 

(4) How can accessions best be utilized to meet but not exceed Officer 
Programmed Authorizations? 

Ideally, the insight gained from answering these research questions will 
quantify the limits of effectiveness of OSAM and lead to its confident use in 
broader applications of manpower modeling. 

D. METHODOLOGY 

The Strategic Assessment Group’s (SAG) work on OSAM has been 
focused on building and using the model; however, this research utilizes data 
farming and experimental design to analyze the model itself. This study was 
conducted within the Simulation Experiments & Efficient Designs (SEED) Center 
for Data Farming ( http.V/harvest.nps.edu) at the Naval Postgraduate School 
(NPS). Data farming relies on an amalgamation of simulation modeling, design 
of experiments, and high-performance computing to “grow” vast amounts of data. 
Applying proven methods of statistical analysis to the farmed data allows the 
opportunity to observe an entire landscape of solutions and enables analysts 
to interpret results, assess model validity, identify and examine outliers, and 
explore otherwise intractable research questions (Horne & Meyer, 2010). 

E. BENEFITS OF RESEARCH 

The ability to data farm with OSAM provides a promising resource for the 
U.S. Navy to gain insight into future manning requirements and more effectively 
project the effects of proposed policies. There is currently a struggle to capture 
the full potential of what OSAM can provide to multiple end users. This study and 
the software written to support it will enable a foundation for future research by 
greatly expanding the capabilities of OSAM and demonstrating how it can be 
improved for use beyond the SAG offices. 
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F. ORGANIZATION OF THESIS 

This thesis is broken into five chapters. Chapter II provides an overview of 
OSAM V3.1.4 and reviews previous studies relating to manpower modeling and 
data farming. Chapter III explores the design of experiments, revisions within 
OSAM, and additional software needed to implement the data farming 
methodology. Chapter IV presents the analysis of the farmed data. Conclusions 
and recommendations for future work are presented in Chapter V. 
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II. BACKGROUND 


In this chapter, we explore the Officer Strategic Analysis Model (OSAM) in 
its configuration as currently employed by Strategic Actions Group (SAG), 
focusing on the aspects of the model studied in depth within this thesis. We also 
review past research and documentation within the manpower and personnel 
modeling field in order to appreciate the foundation upon which this research was 
built. 

A. OFFICER STRATEGIC ANALYSIS MODEL 

The Officer Strategic Analysis Model (OSAM) is an entity-based simulation 
model used to project the officer population across a specified number of years. 
Originally hosted in Visual FoxPro, version 3.1.4 has been translated into Visual 
Basic for Applications and is now hosted entirely within Microsoft Access, along 
with all input and output data. This upgrade also includes a user-friendly 
graphical user interface (GUI) for single-run simulations. 

1. Modeling Parameters 

The foundation for each simulation is the beginning of year inventory 
(BOY) data, which accurately reflects every active duty officer for the current 
fiscal year (FY) without the use of any personally identifiable information (PM). 
Instead, officers are characterized by pay grade, “skill set” (a grouping of like 
designators), and years of commissioned service (YCS), among other attributes. 
Algorithms for losses, promotions, transfers, and accessions are applied 
according to user input in order to forecast the end of year inventory (EOY). If 
there are multiple years in a scenario, the EOY becomes the BOY for the 
subsequent year. Each EOY is characterized in the same manner as the BOY 
and collectively stored in a multi-year summary (MYS) table. The model takes 
approximately 15 minutes to simulate and compile the data for a seven-year 
projection, depending on processor power. An overview of the simulation process 
for each year is shown in Figure 1. 
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Figure 1. Processes in Each Simulation Year 


BOY Inventory 


Losses 


Control Grade 
Promotions 


/ N 

Fully Qualified 
Promotions 

'v_> 

EOY Inventory Accessions 


Transfers 


From Mundy, J. T. (2014). OSAM documentation user manual and technical 
documentation for V3.1.4. Arlington, VA: SAG Corp. 


a. Losses 

Officer losses are applied in three phases for each simulation year. First, 
the model accounts for “natural losses” by applying loss rates from a user- 
specified loss plan within the OSAM database to the BOY inventory (Mundy, 
2014). These plans are typically the actual natural losses from the previous FY 
adjusted by the Strategic Resourcing Branch’s (N100) force planners for any 
foreseeable effects to the loss rate, such as the economy or unemployment rate, 
using the Navy Officer Personnel Planning System Forecasting Model (NFM) (D. 
Clark, personal communication, April 17, 2015). Next, the model will abide by 
programmed U.S. Law and Navy policy limits to “force out” additional losses, for 
example Lieutenant Commanders who have reached 20 YCS and have failed to 
promote. Finally, OSAM will “force out” any additional losses specified by the 
user. User-added losses can be specified by FY, skill set, pay grade, and YCS 
group (Mundy, 2014). 


8 











b. Promotions 

Officer promotions are broken into two categories. First are control grade 
promotions or promotions to pay grades 04 to 06. The Defense Officer 
Personnel Management Act (DOPMA), Department of Defense (DOD), and Navy 
guidelines regulate these promotions by percentage of total officer inventory. 
Officers in the same competitive categories compete for a limited number of 
control grade promotions (e.g., all line officers compete against one another) 
(Schirmer et al., 2006). The user specifies one of four available methods for 
simulating these promotions. For this thesis, “auto-promote” was the only method 
employed. “Auto-promote” simulation restricts the number of promotions to 
remain within the aforementioned guidelines while filling as many vacancies as 
possible (Mundy, 2014). 

The second type of promotion is fully qualified promotions or promotions 
to the 02 and 03 pay grades. Since nearly all officers promote to these pay 
grades, there is no option available to specify any type of user limitations on 
these. 


c. Transfers 

While transfers from a training skill set to a qualified skill set occur simply 
with a pre-programmed time-step, all other transfers follow a user-specified 
transfer plan (Mundy, 2014). This is an essential simulation step as some skill 
sets access officers only by lateral transfers. Additionally, these transfers create 
losses for the parent skill sets. For example, if a surface warfare officer (SWO) 
made a lateral transfer into the human resources (FIR) community, there would 
be two distinct actions with the model: a loss in the SWO community and an 
accession in the HR community. The transfer plans designate the number of 
transfers into skills by pay grade, and OSAM fulfills these transfers according to a 
programmed historical distribution (Mundy, 2014). 
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d. Accessions 

Accessions are the final simulation step for each year. Accessions are 
user-specified by plan or method. The two methods for accessions are 
“unconstrained accessions” and “constrained accessions.” The unconstrained 
accessions method accesses as necessary to meet future requirements, 
regardless of Officer Programmed Authorizations (OPA). If over-manning is a 
concern, constrained accessions can be selected to ensure that EOY does not 
exceed OPA for any year of a simulation sequence (Mundy, 2014). It is important 
to note that only one accession plan can be selected for any run of the 
simulation. Therefore, if an accession plan is selected for multiple years of the 
simulation, the same accession plan, which is most likely tailored to a single FY, 
will be used for every year selected. For example. Figure 2 shows that 
“AccPlan_FY14” is selected and will be utilized by OSAM for FY 2015, FY 2019, 
FY 2020, and FY 2021 simulations. FY 2016 and FY 2017 will utilize the 
unconstrained accessions method, and FY 2018 will utilize the constrained 
accessions method. 


Figure 2. Accession Options in OSAM 



B. LITERATURE REVIEW 

While OSAM is particular to the Navy, the subject of manpower modeling 
has been studied across multiple disciplines and industries. Sibley’s (2012) study 
is the most germane to this thesis, and also provides an excellent review of 
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literature regarding inventory projection, a portion of which has been summarized 
below. A brief review of these works will highlight proven manpower modeling 
methodologies, many of which are observed in the OSAM model, as well as the 
limitations of previous studies. 

1. Manpower Modeling 

Edwards (1983) provides a survey of manpower models and explains the 
three basic elements of all such models: predicting future demand, predicting 
future supply, and assessing policies to reconcile the difference between the two. 
He also outlines the four tenets of a “good” inventory projection model, 
presumably in addition to producing accurate results. First, the model should use 
terms the planner is familiar with. Second, the model results should require 
minimal interpretation. Third, the model’s principles should be understandable to 
a layman. Fourth, the input data should be accurate and available in a timely 
manner. OSAM satisfies all of these requirements; however, there is room for 
improvement with reference data updates. Currently, updates are only provided 
on a yearly basis, despite updated reference data being available on a monthly 
basis. 

Several enlisted personnel models have been developed in keeping with 
the elements described by Edwards. The Accession Supply Costing and 
Requirements (ASCAR) model allows decision makers to see the effects of their 
proposed policy through the time-stepped simulation of size and composition of 
the active duty enlisted force (Collins et al., 1983). However, ASCAR focuses 
mainly on budgetary policies of the proposed inventory size and selection as 
opposed to the changes within the inventory as OSAM does. Fiebrandt (1993) 
developed an Excel spreadsheet model, the U.S. Coast Guard Rating Forecast 
Model that treats many of the personnel flows similarly to OSAM. Unlike OSAM, 
it is only capable of modeling a single designator at once so the impact of a 
policy on the entire Coast Guard enlisted force cannot be easily assessed at 
once. 
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Clark’s (2009) Requirements-Driven Cost-Based Manpower Optimization 
(RCMOP) model has also been studied in depth. This Navy officer model, which 
projects monthly inventory, minimizes unfulfilled manpower requirements while 
remaining within the two-year budget. As with OSAM, inventory is calculated 
based on four processes that model “promotions, accessions, designator 
transfers, and forced and natural losses” (Clark, 2009, p.4). Wheeler (2010) 
continues Clark’s work by developing an extension of the original model, 
RCMOP-2, which doubles the planning horizon and introduces non-linear 
penalties for gapped billets to ensure a dispersion of unfilled billets across 
communities. Both programs are designed with budget optimization as a primary 
goal, whereas OSAM can either be constrained to the budgetary restrictions of 
OPA or allow inventory to develop with different goals in mind. 

2. Design of Experiments 

Erdman (2010) applies experimental design and data farming to the 
Enlisted Specialty model, an Army enlisted manpower model utilizing 859,633 
variables operating under 224,473 constraints. Despite having few modeling 
processes in common with OSAM, the successful simulation analysis of the 
farmed data of an extremely complex model speaks volumes to the methodology 
employed. The results of this analysis led to changes in the model coefficients as 
implemented by the Army manpower planners, which reduced the misalignment 
of soldiers to billets by over 15%, or approximately 8,355 soldiers per month. 

3. OSAM Studies 

Sibley’s work (2012), which provides the groundwork for this thesis, 
analyzes a previous version of the OSAM model using design of experiments 
and data farming to explore the lateral transfer and loss functions in depth. 
Although limited in scope to only four OSAM skill sets over a five-year projection, 
the advanced data analysis applied in Sibley’s study resulted in loss adjustment 
factors superior to those previously in use by N100 planners. Following Sibley’s 
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recommendations for future work, this study varies a different set of parameters 
over the entire officer inventory for a longer projection of seven years. 

DeHollan (2015) has continued to analyze losses in OSAM. Like this 
work, he has incorporated stochastic variation in order to data farm over a design 
of experiments. However, where this study focuses on the interaction of multiple 
parameters, DeHollan looks in depth at the effect of varying periods of poor 
retention on a portion of the Unrestricted Line (URL) officer community. His 
resulting model provides an alternative predictive capability for specific 
designators. 

Since OSAM is currently used by analysts at SAG, there have been 
several unpublished studies utilizing the same version of the model. In these 
analyses, OSAM is employed to study a proposed policy’s effect on a single skill 
set community, usually varying a specific parameter at a time and plotting the 
resulting end strength of the community (W. Corley, personal communication. 
May 22, 2015). This thesis’s contribution lies in implementing a stochastically- 
driven model to analyze the effect of inherent variation on officer end strength 
and the application of data farming to explore where manipulated parameters 
may have confounded effects. 
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III. METHODOLOGY AND IMPLEMENTATION 


In this chapter, we explore the resources used to make the leap from 
using the basic Officer Strategic Analysis Model (OSAM) graphical user interface 
(GUI) to run a single simulation to implementing a design of experiments (DOE) 
and data farming techniques necessary to grow big data. The data farming loop, 
as depicted in Figure 3, guides our implementation. The left side of the data 
farming loop depicts the scenario, or distillation, building loop. Since this research 
focuses on a pre-built model, we are left only with building the right hand side of 
the loop. This chapter focuses on defining the design space and executing the 
design matrix simulations. 

Figure 3. The Data Farming Iterative Process 



Data Farming Loop 

From Florne, G. E., & Schwierz, K. (2008). Data farming around the world 
overview. In Proceedings of the 2008 Winter Simulation Conference, 264-285. 
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A. DESIGN SPACE 

1. Experiment Parameters 

The first step in an effective design of experiments is to decide which 
model variables or parameters are of greatest interest to the policy we are 
exploring as well as any interacting parameters that may represent the modeling 
environment (Horne & Meyer, 2004). In this case, we would like to compare the 
merits of possible Enhanced Probationary Officer Continuation and Re¬ 
designation (EPOCR) policies (Kleijnen et al., 2005). We need to manipulate the 
number of SWOs sent to Probationary Officer Continuation and Re-designation 
(POCR) boards and the officer accession options utilized to find the combination 
that best suits our purpose of reconciling officer inventory to Officer Programmed 
Authorizations (OPA). If enacted, the chosen policy would need to perform 
adequately across a span of possible natural losses, since we are unable to 
predict or control the actual natural losses we will encounter in the future. These 
conditions lead to the inclusion of the parameters listed in Table 1. 


Table 1. Table of Selected Factors and Acceptable Ranges 

within OSAM 


Factor 

Input Range 

Controllable Factors 

Forced SWO Losses 
(through POCR) 

0+ 

Accession Options 

Accession Plans (FY11-14), 
Unconstrained Accessions, 
Constrained Accessions 

Uncontrollable Factors 

Natural Loss Rate 

Loss Plans (FY06 - FY14) 
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2. Policy Values 

The next step to complete the design of experiments is to determine the 
constraints of the experiment and translate the proposed policy into parameter 
values. The experiment will be conducted over a seven-year projection utilizing 
all accession options, but the only plan used will be the most recent available, 
that of FY14. The natural loss rate will be represented through a low-loss plan, a 
high-loss plan, and the most recent loss plan. According to the Officer Strength 
Planner, the FY13 loss plan represents the lowest loss rates likely to be repeated 
while the FY09 loss plan represents the highest loss rates likely to be seen in the 
near future (P. Saluke, personal communication. May 28, 2015). Therefore, these 
plans will form the bounds on loss rates, and the FY14 plan will represent the 
predicted loss rate. The values for the POOR losses will represent POOR as 
currently implemented with no additional losses, EPOCR as analyzed utilizing 
other models with 120 losses, and an upper bound of 200 losses based on the 
results of individually run screening experiments (P. Saluke, personal 
communication, June 19, 2015). 


Table 2. Appropriate Values of Factors 


Factor 

Value 1 

Value 2 

Value 3 

Controllable Factors 
(coded values) 

Forced SWO 
Losses 

(through POOR) 

0 

120 

200 

Accession 

Options 

Accession Plan 

(0) 

Unconstrained 

Accessions 

(1) 

Constrained 

Accessions 

(2) 

Uncontrollable Factors 
(coded values) 

Natural Loss 
Rate 

FY14 Predicted 
(0) 

FY13 Low Loss 
(-1) 

FY09 Fligh Loss 
(1) 


17 

















3. Experimental Design 

An experimental design eliminates the need for trial and error and guides 
in the methodical approach to observing the effects of changing variables. 
Precisely stated, it is a combination of factors and values, where the columns 
correspond to each factor varied during the experiment and the rows are 
comprised of a corresponding value for each factor (Kleijnen et al., 2005). Each 
unique row represents a design point. The total number of design points possible 
is a product of the total number of values considered for each factor. In this 
experiment with three factors, each with three possible values, the maximum 
number of design points is 3^ or 27. A full factorial design is the combination of all 
27 design points, as shown in Table 3. 

Given the relatively short amount of time it takes to run a single design 
point and the vast Simulation Experiments & Efficient Designs (SEED) Center 
resources that lend themselves to automation and compilation of large data sets, 
there is really no need to compress the design any further. Additionally, full 
factorial designs are inherently straightforward and have several benefits for data 
analysis. Full factorial designs can be analyzed to identify all interactions, so no 
confounded effects will remain unexplained (Sanchez & Wan, 2012). 
Furthermore, full factorial designs are also orthogonal, guaranteeing that factors 
are uncorrelated and further simplifying the upcoming data analysis (Kleijnen et 
al., 2005). This can be seen in Figure 4. 
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Table 3. Full Factorial Design of Experiments (Uncoded) 


DESIGN 

POINT 

ACCESSION OPTIONS 

LOSS PLAN 

POOR 

1 

Accession Plan 

LossRates_FY13 

200 

2 

Accession Plan 

LossRates_FY14 

200 

3 

Accession Plan 

LossRates_FY09 

200 

4 

Accession Plan 

LossRates_FY13 

120 

5 

Accession Plan 

LossRates_FY14 

120 

6 

Accession Plan 

LossRates_FY09 

120 

7 

Accession Plan 

LossRates_FY13 

0 

8 

Accession Plan 

LossRates_FY14 

0 

9 

Accession Plan 

LossRates_FY09 

0 

10 

Constrained Accessions 

LossRates_FY13 

200 

11 

Constrained Accessions 

LossRates_FY14 

200 

12 

Constrained Accessions 

LossRates_FY09 

200 

13 

Constrained Accessions 

LossRates_FY13 

120 

14 

Constrained Accessions 

LossRates_FY14 

120 

15 

Constrained Accessions 

LossRates_FY09 

120 

16 

Constrained Accessions 

LossRates_FY13 

0 

17 

Constrained Accessions 

LossRates_FY14 

0 

18 

Constrained Accessions 

LossRates_FY09 

0 

19 

Unconstrained Accessions 

LossRates_FY13 

200 

20 

Unconstrained Accessions 

LossRates_FY14 

200 

21 

Unconstrained Accessions 

LossRates_FY09 

200 

22 

Unconstrained Accessions 

LossRates_FY13 

120 

23 

Unconstrained Accessions 

LossRates_FY14 

120 

24 

Unconstrained Accessions 

LossRates_FY09 

120 

25 

Unconstrained Accessions 

LossRates_FY13 

0 

26 

Unconstrained Accessions 

LossRates_FY14 

0 

27 

Unconstrained Accessions 

LossRates_FY09 

0 
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Figure 4. Scatterplot Matrix for Full Factorial Coded Design 



B. REPLICATIONS 

1. OSAM Modifications 

A replication is the repetition of the entire design matrix (Sanchez & Wan, 

2012). Multiple replications are needed to infer statistically significant patterns 

and variations in the resulting data. Since the current version of OSAM functions 

as a deterministic model, if the design of experiments were to be implemented, 

we would end up with only 27 unique observations no matter how many 

replications were made. This is because although OSAM v3.1.4 was designed to 

be a stochastic model, the random numbers are all hard-coded in Visual Basic 
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for Applications, acting as constants. Therefore, there is no variation among 
replications of the same design point. Mr. Steve Upton, a SEED Center Research 
Associate, corrected this by replacing the “random” number constants with a 
Mersenne Twister random number generator (Matsumoto & Nishimura, 1998). 
Following the practice of “good programming” described by Kleijnen et al. (2005), 
the Mersenne Twister generator requires an external random seed argument. In 
order to pass this argument and the other parameters that would normally be 
selected within the GUI, Upton stripped OSAM of the GUI interface, allowing 
OSAM to receive inputs as a list of arguments instead (Upton, 2015, p. 1). This 
relieved the need for human interaction with every run as an initiation file can 
now provide all the necessary information for a simulation. 

2. Support Software 

Upton also created the software OSAMRunner using C++. OSAMRunner 
replaces the scenario editor that was originally located in the GUI, allowing the 
user to run one or more replications of an OSAM scenario via the command line 
interface. OSAMRunner parses and verifies command line arguments and then 
calls the modified OSAM program and specified initialization file. Once a 
replication scenario is complete, OSAMRunner collects the output file to a 
“tables” folder where it is stored as a CSV-formatted file. It then loops based on 
the number of seeds provided, all without further user interaction (S. Upton, 
personal communication, February 13, 2015). This effectively extends the 
possible number of scenarios simulated in a single user session to the number of 
seeds provided by the user at one time. 

OSAMFarmer, another C++ program created by Upton, works in 
conjunction with OSAMRunner by compiling the initialization files needed to 
translate each design point to an OSAM scenario. OSAMFarmer is designed to 
create multiple variations based on an experimental design. For each excursion 
created by OSAM Farmer, OSAMRunner will complete the specified replications 
for each random number seed provided. 
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3. Cluster Computing 

Running the modified OSAM version using the support software described 
above over all 27 design points of the design matrix would take approximately 
6.75 hours on a standard desktop computer. Doing so for 100 replications would 
take nearly a month, hardly a reasonable time period when looking to quickly 
gain insight into a pressing issue. However, splitting the replications among 
several processors reduces the computing time. Using existing SEED Center 
software, Upton distributed the replications across 20 SEED Center processors 
and was able to complete 100 replications of all 27 design points in 41.4 hours 
(S. Upton, personal communication, July 30, 2015). This equates to 153 more 
hours of actual computing time because the replications were run in batches of 
ten in order to compile data and clean up the one terabyte of disk space between 
batches. If storage had not been an issue, all replications could have been 
completed in slightly over a day. 
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IV. ANALYSIS OF EXPERIMENTAL RESULTS 


This chapter examines the results derived by data farming the Officer 
Strategic Analysis Model (OSAM). We use a variety of graphical and statistical 
methods to scrutinize this data. All figures and analyses in this chapter were 
produced using JMP Pro Version 12.1.0 (http://www.jmp.com/en_us/home.html), 
an interactive statistical analysis tool developed by the SAS Institute Inc. (2015). 

A. OVERVIEW OF RESULTS 

The selected data compiled from all OSAM simulation replications yields 
a 450MB comma separate value file comprised of 6,400,000 unique 
observations, each containing values for design point, accession option, loss 
plan. Probationary Officer Continuation and Re-designation (POOR) losses, 
seed, fiscal year (FY), grade, skill set, and end of year inventory (EOY). A portion 
of this data is shown in Table 4. This raw data is manipulated in JMP to produce 
summary tables and various subsets of data in order to conduct various 
analyses. The majority of this study utilizes values for the total EOY across all 
skill sets and grades, called end strength. The raw data has been summed over 
design point, accession option, loss plan, POOR, seed, and FY to generate the 
end-strength values, as seen in Table 5. 


Table 4. Sample of Raw Data Format 


Design 

Point 


Accession 

Option 


Loss Plan 

POCR 

seed 

Fiscal 

Year 

Grade 

Skill 

End of Year 
Inventory 


1 


0 

-1 

200 

16296558 

2015 

1 

111X - Surface Warfare 

225 


1 


0 

-1 

200 

16296558 

2015 

1 

112x - Submarine Warfare 

56 


1 


0 

-1 

200 

16296558 

2015 

1 

113x - Special Warfare 

57 


1 


0 

-1 

200 

16296558 

2015 

1 

114x - Special Operations 

8 
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Table 5. Sample of Summary Data Format 


Design 

Point 



Accession 

Option 

Loss Plan 

POOR 

seed 

Fiscal 

Year 

N Rows 

End Strength 


1 

0 


-1 

200 

352665 

2015 

344 

52428 


1 

0 


-1 

200 

352665 

2016 

339 

52617 


1 

0 


-1 

200 

352665 

2017 

332 

52819 


1 

0 


-1 

200 

352665 

2018 

325 

52954 


B. STOCHASTIC VARIATION 

Before variations across the DOE can be explored, we must first assess 
the stochastic variation induced by implementing random seeds in OSAM. Every 
replication utilizes a different random seed, resulting in one hundred random 
seeds in total. In order to visualize the stochastic variation, the end strength is 
plotted by FY for each design point. The resulting plots, located in Appendix A, 
take on three distinct shapes. 

For this discussion, we assign three design points, one per end-strength 
shape, as “base cases.” Each base case follows the predicted loss plan with no 
additional forced losses via POOR. Therefore, only the accession option varies. 
The base cases translate to design point 8 for accession plan, design point 17 for 
constrained accessions, and design point 26 for unconstrained accessions. Each 
base case end-strength plot is shown in Figure 5. Although variation increases 
with each additional simulation year, overall, there is very little variation in end 
strength among seeds. The largest difference for all design points occurs in the 
last simulation year, FY 2021. For design point 17, this range is still less than 
0.25% of the mean end strength for that design point. The summary statistics are 
reported in Table 6 for each base case and in Appendix B for all design points. 
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Figure 5. 


Stochastic Variation in End Strength for Base Cases 
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Table 6. Summary Statistics of End Strength for Base Cases 


Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 
(as % 
Mean) 

Design Point 8 

2015 

52502 

52509 

52511 

52511.56 

52514 

52520 

0.034 

2016 

52746 

52753 

52757 

52757.42 

52761 

52780 

0.064 

2017 

53002 

53014.25 

53019 

53019.93 

53025 

53042 

0.075 

2018 

53219 

53232 

53238 

53239.33 

53246.75 

53265 

0.086 

2019 

53316 

53338 

53351 

53351.99 

53365.75 

53394 

0.146 

2020 

53404 

53434 

53447 

53449.03 

53466.75 

53498 

0.176 

2021 

53512 

53544.25 

53555.5 

53557.96 

53570 

53607 

0.177 

Design Point 17 

2015 

52104 

52117 

52123 

52121.98 

52126 

52134 

0.058 

2016 

51977 

51997 

52004 

52004.23 

52013 

52028 

0.098 

2017 

51886 

51909.5 

51917.5 

51916.88 

51927 

51947 

0.117 

2018 

51756 

51782.25 

51792 

51794.37 

51807 

51837 

0.156 

2019 

51546 

51573 

51591 

51590.36 

51604 

51646 

0.194 

2020 

51351 

51394.25 

51408 

51407.52 

51421.75 

51451 

0.195 

2021 

51202 

51250.25 

51266.5 

51264.39 

51281.75 

51328 

0.246 

Design Point 26 

2015 

51349 

51351 

51352 

51351.96 

51353 

51355 

0.012 

2016 

53126 

53133 

53134 

53134.24 

53135 

53138 

0.023 

2017 

50338 

50339.25 

50340 

50340.45 

50341 

50344 

0.012 

2018 

50679 

50681 

50682 

50682.23 

50683 

50686 

0.014 

2019 

50735 

50738 

50739 

50739.31 

50740 

50743 

0.016 

2020 

50735 

50737 

50738 

50737.84 

50739 

50742 

0.014 

2021 

50734 

50737 

50738 

50738.34 

50739 

50744 

0.020 
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C. GRAPHICAL ANALYSIS 

Although JMP offers a wide selection of statistical analysis tools, a general 
understanding of the DOE can often be obtained by graphs plotted across the 
spectrum of factor values. These graphs can identify practically significant factors 
and provide a qualitative understanding of OSAM’s behavior. 

It is natural to begin by exploring the influence of each factor of the DOE 
separately. Figure 6 displays the mean end strengths by fiscal year as affected 
by changes in a single factor through a series of subplots. Again, the shape of 
the curve is clearly related to the accession option. We see this by analyzing 
different color curves within the same subplot. Studying the same color curve 
vertically across the subplots shows the effect of varying loss plans. We note that 
loss plans cause negligible variation when combined with accession options 1 
and 2. Examining the same color curve horizontally across subplots reveals the 
effect of POOR losses. POOR losses slightly decrease the slope of the curves for 
accession options 0 and 1, resulting in increasingly smaller end strengths. 
However, POOR losses appear to have no affect on accession option 2 curves. 
Indeed, accession option 2 appears to be independent of both POOR losses and 
loss plans. 
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Figure 6. DOE Factors’ Effect on End Strength 


End Strength by Fiscal Year 

POOR 

0 120 200 



FY 


Accession Option 

- 2 

—- OPA 


Translating these observations from the coded DOE factors into a 
manpower context provides insight into the behavior of the personnel plans and 
force-shaping policies modeled by OSAM. Officer Programmed Authorizations 
(OPA) are also plotted in Figure 6 to provide a reference for desired end 
strength. It is apparent that, as the last step of the simulation process, the 
accession option greatly affects the end strength and can even compensate for 
the applied loss plan. We also find that despite a caution for possible over¬ 
manning within the OSAM documentation, the unconstrained accessions method 
(1) consistently results in the lowest end strengths across all high loss, both 
natural and forced, scenarios (Mundy, 2014). In fact, there is ample risk of falling 
below OPA with this accession option should losses increase above the 
predicted plan even before implementing any additional type of POOR policy. 
This counterintuitive result was verified by running several replications by hand in 
the original version of OSAM, which yielded similar values. 
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In order to increase visibility of the variability in end strength induced 
across loss plans, we will analyze end-strength distributions by accession option 
utilizing a boxplot display. A key to JMP’s standard boxplot diagram is shown in 
Figure 7. 


Figure 7. JMP Boxplot Structure 



median 

-sample 

value 

^75% quantile 
or3rd quartile 

25% quantile 
or 1st quartile 


From SAS Institute Inc. (2015). Basic analysis JMP 12. Cary, NC: SAS Institute 
Inc. 


Technically, any variability induced by POOR losses would also be 
included in this rendering of end-strength variation, but as seen from Figure 6, 
that tends to be consistent across all loss plans and imparts a small fraction of 
the variation compared to loss plan. From Figure 8, we see that variation from 
loss plan is reduced by over 50% when the unconstrained accessions method (1) 
is chosen over an accession plan (0). When the constrained accessions method 
(2) is chosen, there is essentially no variation of end strength, no matter which 
loss plan is utilized. 
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Figure 8. Variability Induced by Loss Plan 
End Strength by Fiscal Year 


Accession Option 



Fiscal Year 


Next, we examine the effect of POOR losses on the Surface Warfare 
Community. Although the effect of POOR may be slight when observing the 
overall officer end strength, the results on the targeted community can be severe. 
In Figure 9, the SWO EOY is plotted by FY for each grade. It is important to 
inspect the community inventory at this detailed level in order to ensure that the 
targeted year group still has sufficient inventory remaining to promote and fill the 
higher-ranking grades. Although the SWO 01- 03 inventory varies by accession 
option, there is no variation of higher ranks across differing POOR values. This 
confirms that there will not be a shortage of junior officers to promote. 

Another consideration when dealing with POOR is communities that 
source their inventory from lateral transfers. The SWO community is the largest 
source for lateral transfers so a dip in inventory could have second- and third- 
order effects in another skill set. In order to confirm that POOR losses were not 
affecting any other community, all skill sets were plotted in the same manner as 
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seen in Figure 9. No other skill set showed any sort of EOY deviation across 
POOR loss values. 


Figure 9. POOR Effect on Surface Warfare Community by Grade 
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D. STATISTICAL ANALYSIS 

In addition to providing insight about the behavior and interactions of the 
DOE factors, the resulting data can also be analyzed to determine statistical 
significance. In order to evaluate and predict the effects of the DOE factors on 
OSAM’s inventory projections, we must first identify a measure of effectiveness. 
In this analysis, we are concerned with estimating variations from the mean end 
strength. A quantitative measure of that variability, or the deviation from the 
mean, is the standard deviation. 
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While it is easy to infer from a visual inspection of the preceding plots that 
accession option and fiscal year contribute to the standard deviation, we will take 
a step back and use statistical methods to assess all simulation factors. We 
begin by averaging the standard deviation at every design point by fiscal year 
and using JMP to plot and analyze the results. Figure 10 graphically displays the 
standard deviations as well as means diamonds for each design point. The 
diamond’s height represents the 95% confidence interval for the mean standard 
deviation for each specific design point, and the center bar displays the group 
mean (JMP, 2015). 

Figure 10. One-way Analysis of Standard Deviation of the 
End Strength by Design Point 



A one-way analysis of variance (ANOVA) can be conducted to determine 
whether the difference in means is similar to what would have been expected by 
chance or statistically significant (Faraway, 2005). The null hypothesis for this 
analysis is that every design point results in the same standard deviation. As is 
expected from the graph, this hypothesis is rejected. The results of the ANOVA 
analysis in Figure 11 show that the p-value is less than 0.0001. Even with error 
terms that are not perfectly normal in distribution and nonconstant variances 
across design points, a p-value estimated to be that small reveals that the 
differences are statistically significant as well (Faraway, 2005). 
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Figure 11. Standard Deviation by Design Point ANOVA results 


Sum of 


Source 

DF 

Squares 

Mean Square 

F Ratio 

Prob > F 

Decision Point 

26 

6467.144 

248.736 

10.0430 

<.0001* 

Error 

162 

4012.268 

24.767 



C. Total 

188 

10479.412 





The next step in our analysis is to detect which differences are statistically 
significant. Faraway states that the Tukey, also referred to as the Tukey-Kramer, 
honest significant difference (FISD) test is appropriate to compare all pairwise 
differences in equal-sized samples (2005). Utilizing this test on our data in JMP 
results in the large circles on the right in Figure 10. These circles represent 95% 
confidence intervals and their intersections with other circles depict significant 
differences as shown in Figure 12. 


Figure 12. Confidence Interval Circles Angles of Intersection 

and Significance 

angle greater angle equal to angle less than 

than 90 degrees 90 degrees 90 degrees 



not significantly 
different 



borderline 

significantly 

different 



significantly 

different 


From SAS Institute Inc. (2015). Basic analysis JMP 12. Cary, NC: SAS Institute 
Inc. 


In addition to the confidence interval circles, which highlight the labels of 
other design points that are not significantly different when selected, JMP also 
displays a connecting letters report, shown in Figure 13, as part of the Tukey- 
Kramer FISD test results. This report lists the design points under “levels” as well 
as their means. Between the two columns are letters. Two levels have been 
determined significantly different if they are not connected by the same letter. 
Through JMP’s interactive display and connecting letters report, we can confirm 
that design points 19 through 27 are statistically different from most other design 
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points. These data points correspond to accession option 2, which we have 
already seen produce very different end-strength values with less variation than 
the other two accession options. 


Figure 13. Standard Deviation by Design Point 
Connecting Letters Report 


Level 


Mean 

13 

A 

14.344529 

16 

A 

14.127116 

15 

A 

14.118529 

10 

A 

13.611234 

5 

A 

13.346556 

1 

A 

12.273249 

7 

A 

12.116725 

8 

A 

12.003696 

2 

A 

11.984277 

3 

A B 

11.493515 

9 

A B 

11.092679 

6 

A B 

11.057775 

4 

A B 

10.049009 

21 

B 

1.707909 

24 

B 

1.621270 

23 

B 

1.601490 

20 

B 

1.599568 

25 

B 

1.574443 

26 

B 

1.543125 

27 

B 

1.510071 

19 

B 

1.508218 

22 

B 

1.505326 


We continue exploring the standard deviation by conducting one-way 
ANOVA and Tukey-Kramer HSD tests individually on the DOE factors to 
determine which factors contribute towards the variance detected at the design 
point level. As seen in Figure 14, the accession option contributes heavily to the 
variance, while the loss plan and POOR losses are not significant. These visual 
assessments can be confirmed by the ANOVA results, which report a p-value of 
less than 0.0001 for the accession option, and connecting letters reports found in 
Appendix C. This confirms what we have already seen to some extent in the 
differences between the base case plots. 
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Figure 14. One-way Analysis of Standard Deviation of the End Strength 

by Design of Experiment Factors 



0.05 


25 


s 20 
55 


c 15 
s 
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0.05 



POOR 


Tukey-Kramer 

0.05 


Although not technically a factor in the DOE, the plots of our data, like 
Figure 15, have indicated that variance increases almost directly with fiscal year. 
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We investigate this relationship with a one-way ANOVA that reports a p-value of 
less than 0.0001. The Tukey-Kramer connecting letters report shows adjacent 
years to be similar with significant differences at two-year intervals and greater. 

Figure 15. One-way Analysis of Standard Deviation of the End Strength 

by Fiscal Year 



0.05 


E. METAMODEL 

Flaving identified independent factors’ influences on the standard deviation 
of end strength, we must now investigate their interactions and cumulative 
effects. The amalgamation of all factors’ contributions toward the standard 
deviation will help us build a metamodel to better understand the framework and 
relationships modeled by OSAM. 

1. Multiple Linear Regression 

JMP provides a powerful interactive regression tool capable of utilizing 
stepwise regression with a designated direction and stopping rule to formulate 
the best model for the specified response and explanatory variables. The 
response variable of interest continues to be the standard deviation of end 
strength. Possible explanatory variables include loss plan, accession option, 
POOR losses, FY, as well as all interactions and quadratic terms. Often, a model 

that is powerful, but is difficult to interpret, can be constructed. In order to avoid 
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this and select the simplest adequate model, we specify the Akaike Information 
Criterion (AlC) as the stopping rule. AlC scores candidate models based on their 
log-likelihood values while penalizing each additional term. The resulting model 
has a high R-squared of 0.97 and performed well in prediction and cross 
validation. However, most of the prediction estimate is based on an interaction of 
accession options. While this may be numerically accurate and powerful, it is not 
representative of the actual behavior we are modeling, as accession options are 
not additive. Therefore, we reject the regression model. Several other models 
were attempted using different combinations of explanatory variables, stepwise 
directions, and stopping rules; however all were heavily dependent on the 
accession option interaction term and also rejected. 

2. Regression Tree 

JMP also provides an interactive nonparametric partitioning platform that 
allows analysis via regression trees. The software utilizes an algorithm that 
recursively bins and averages predictor values in order to partition the sample on 
the factor and value that reduces the total sum of squares to the greatest extent. 
In other words, JMP “partitions the data to provide the most explanatory power 
for a performance measure of interest” (Kleijnen et al., 2005). A key to the 
information displayed in JMP’s regression tree diagram is shown in Figure 16. 


Figure 16. JMP Regression Tree Structure 
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Unlike the stepwise regression method, there is no option to set an ending 
rule for partitioning. However, JMP provides the R-squared and AlC value so that 
the analyst can decide when the tree is adequately grown. There is also the 
option of looking at the leaves of the lowest branches to ensure the response 
variable, in this case the standard deviation, is within acceptable limits. For 
instance, if the goal of the model were to reduce standard deviation of the 
response to below 5, the regression tree model in Figure 17 would be adequate 
after just two splits. This tree was split until the AlC began to increase, at which 
point the last branches were pruned back so that the resulting tree had a 
combination of one of the highest R-squared values and the lowest AlC. This is 
confirmed in the split history report graphed in Figure 18. 

This method of analysis proves to be especially convenient to explain the 
standard deviation of end strength. The 14-split partition tree model shown in 
Figure 17 took only minutes to produce and conveys the results in an accurate 
and straightforward method that a non-analyst would have little trouble 
understanding. As expected, the first split occurs by accession option and 
reduces the range of standard deviations across all grouped rows by nearly 11. 
The second split occurs according to fiscal year, and again reduces the variation 
by nearly ten. All remaining splits have a much smaller impact, the largest being 
approximately 50% of the first two splits, on the remaining variation within 
groups. 

With an R-squared of 0.974 the model explains over 97% of the variance 
in standard deviation. Before partitioning, the data was split into a training set 
with 80% of the original data and a test set made up of the remaining data. 
Validating the tree with the test set results in a negligibly lower R-squared of 
0.971, suggesting the model is correct and neither under- nor over-fit. This is also 
displayed in Figure 19 by graphing the actual residuals by the predicted residuals 
for both the training and test datasets. 
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Figure 17. Standard Deviation Partition Tree 


Number 

RSquare RMSE N of Splits AlCc 
Training 0.974 1.1892492 164 14 557.962 

Vaiidation 0.971 1.3115591 25 


All Rows 

Count 164 LogWorth Difference 

Mean 9.1197914 55.549382 11.7689 
Std Dev 7.3459658 


Accession Option(2) 
Count 59 

Mean 1.5848527 

Std Dev 0.2132381 


Accession Optlon(0,1) 

Count 105 LogWorth Difference 

Mean 13.353709 51.755962 9.54071 
Std Dev 5.8518466 


I 7 

Rscal Year(2015, 2016.2017. 

2018) 

Count 61 LogWorth Difference 

Mean 9.3556963 12.457361 4.8136 

Std Dev 3.6121605 

I 


Fiscal Year(2019.2020, 2021) 

Count 44 LogWorth Difference 

Mean 18.896409 4.2778574 3.5781 

Std Dev 3.2412588 


Fiscal Year(2015,2016) 

Count 30 LogWorth Difference 

Mean 6.9094417 6.3153286 3.51968 
Std Dev 2.5910664 


Fiscal Year(2017.2018) 

Count 31 LogWorth Difference 

Mean 11.723039 12.224231 4.43848 
Std Dev 2.7982642 


Fiscal Year(2019, 2020) 

Count 29 LogWorth Difference 

Mean 17.676604 3.5395912 3.05861 
Std Dev 2.6831243 


Fiscal Year(2021) 

Count 15 LogWorth Difference 

Mean 21.254699 1.5473245 3.28345 
Std Dev 2.9712725 


Fiscal Year(2015) 

Count 13 LogWorth Difference 

Mean 4.9149545 18.14916 2.75264 
Std Dev 1.4721023 


Fiscal Year(2016) 

Count 17 LogWorth Difference 

Mean 8.4346378 10.777324 3.97226 
Std Dev 2.1998663 


Accession Option(O) 

Count 17 LogWorth Difference 

Mean 9.7185632 3.7444296 2.24733 
Std Dev 1.5208144 


Accession Optlon(l) 

Count 14 LogWorth Difference 

Mean 14.157046 2.9157185 2.7421 

Std Dev 1.8944509 


Accession Option(0) 

Count 13 

Mean 15.989097 
Std Dev 2.1732027 


Accession Option(l) 

Count 16 LogWorth Difference 

Mean 19.047704 1.5345816 2.73516 
Std Dev 2.2774867 


Accession Option(0) 
Count 8 

Mean 19.72242 

Std Dev 2.3353077 


Accession Optlon(l) 

Count 7 

Mean 23.005875 
Std Dev 2.743886 


Accession Option(O) 
Count 6 

Mean 3.4327638 

Std Dev 0.166562 


Accession Option(1) 

Count 7 

Mean 6.1854036 
Std Dev 0.4807436 


Accession Option(0) 

Count 8 

Mean 6.3316771 
Std Dev 0.4849735 


Accession Option(l) 
Count 9 

Mean 10.303936 

Std Dev 1.0580736 


Fiscal Year(2017) 
Count 9 

Mean 8.6609966 
Std Dev 0.7971539 


Fiscal Year(2018) 
Count 8 

Mean 10.908326 
Std Dev 1.2266147 


Fiscal Year(2017) Rscal Year(20ie) 
Count 7 Count 7 

Mean 12.785994 Mean 15.528098 
Std Dev 0.9722288 Std Dev 1.5635393 


LossPian(-1) Loss Plan(1,0) 

Count 6 Count 10 LogWorth Difference 

Mean 17.338228 Mean 20.073389 0.6983602 1.84886 

Std Dev 1.221263 Std Dev 2.1677593 


Fiscal Year(2019) 
Count 5 

Mean 19.148961 
Std Dev 2.0963636 


Fiscal Year(2020) 
Count 5 

Mean 20.997817 
Std Dev 2.0104718 
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Figure 18. Regression Tree Split History Report 



Validation Data in Red 


Figure 19. Actual versus Predicted Residuals for Training and Test Sets 


Training Set 



Validation Set 
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V. CONCLUSIONS AND RECOMMENDATIONS 


This thesis corroborates Sibley’s (2012) findings and confirms the value of 
data farming with the Officer Strategic Analysis Model (OSAM). Every distinct 
design of experiments offers the possibility of new insights, refinement of the 
current modeling process, and even additional uses for OSAM. All of these would 
benefit the Navy by gaining maximum return of investment from a pre-existing 
model and minimizing the fiscal and capabilities risk inherent to manpower 
planning. 

A. STOCHASTIC VARIATION IN THE OFFICER STRATEGIC ANALYSIS 

MODEL 

OSAM has been modified to run stochastically and the results offer 
context to the modeling output. The regression tree model shows the different 
standard deviations of end strength associated with varying modeling 
parameters. Analysts should be aware that these standard deviations can be of 
quite different orders of magnitude. 

B. VARIATION OF LOSS RATES ON END STRENGTH 

Surprisingly, we find that variation of loss rates has little to no effect on 
end strength unless manipulated in conjunction with an accession plan. The large 
flexibility of the other accession methods can greatly compensate for any effect 
created by the loss rate alone. 

C. IMPACT OF PROBATIONARY OFFICER CONTINUATION AND RE¬ 
DESIGNATION BOARD LOSSES 

Because of the unusual responses created by the constrained and 
unconstrained accessions methods. Probationary Officer Continuation and Re¬ 
designation (POOR) board losses and the proposed enhanced POOR (EPOCR) 
policy could only be adequately observed in conjunction with the designated 
accession plan. Even with the most aggressive EPOCR policy losses, officer 
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over-execution would remain constant at predicted loss rates or continue to 
increase if loss rates are lower than predicted. If loss rates are greater than 
predicted, additional EPOCR losses could cause an end strength significantly 
below OPA. Should this policy be implemented, natural loss rates will have to be 
carefully monitored to avoid over-correction of end strength. 

D. ACCESSION OPTION UTILIZATION 

It is evident throughout every manipulation of the results that the 
constrained accessions method is the accession option that most easily meets 
OPA. Modeling with this parameter shows an immediate return to OPA and 
eliminates the need for force-shaping policy. However, an in-depth study into the 
accession values created by this option and their future repercussions is required 
before any type of implementation. 

E. RECOMMENDATIONS FOR FUTURE STUDY 

Although this study was able to provide definitive answers for all proposed 
research questions, a better understanding of the model through in-depth 
analysis has exposed additional areas for further analysis. 

1. Incremental Loss and Accession Plan Variations 

This research was limited by only studying the interactions between pre¬ 
existing loss and accession plans. Even so, a clear relationship was revealed 
where accessions can completely conceal loss effects. Implementing a larger 
design of experiments over incremental changes in each type of plan would help 
refine this relationship, ideally identifying a tipping point where losses become 
too extreme for accessions to adapt to the goal end strength. 

2. Accession Plan Modeling Parameters 

In keeping with the first recommendation, a deeper understanding of 
accession modeling in OSAM would be valuable on multiple levels. Clearly, there 
is some misunderstanding regarding unconstrained accessions that needs to be 
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investigated. Either the documentation erroneously warns of over-manning or 
there is a portion of OSAM that is being simulated incorrectly. 

Additionally, if accession methods can entirely absorb the affects of loss 
plans, perhaps this suggests new uses for OSAM to quickly create flexible 
accession plans to respond to unforeseen loss scenarios. In its current 
configuration, the accessions created by the constrained accessions method 
could be retained to provide a range of requirements needed to meet OPA. 
However, considerable deliberation must be given to how the model is treating 
accessions. Is it possible for manpower policy and plans to be as dynamic as 
currently modeled in OSAM? If not, is there a way to make this parameter more 
realistic? 

3. End-strength Anomaly in FY 2018 

A final peculiarity is the change in variation occurring in FY 2018. While it 
is visibly detectable in most of the design point plots contained in Appendix A, it 
is especially apparent in Design Point 2, as seen in Figure 20. Investigation into 
this anomaly may reveal errors or unusual relationships within OSAM. 

Figure 20. End-strength Variation by Fiscal Year for Design Point 2 
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APPENDIX A. STOCHASTIC VARIATION IN END STRENGTH 
ACROSS ALL DESIGN POINTS 



Design Point s 1 
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APPENDIX B. SUMMARY STATISTICS OF END STRENGTH 

FOR BASE CASES 


Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

Desian Point 1 

2015 

52424 

52430 

52432 

52432.0 

52435 

52438 

0.027 

2016 

52598 

52610 

52615 

52614.3 

52619 

52630 

0.061 

2017 

52789 

52813 

52820. 

52820.0 

52826. 

52841 

0.098 

2018 

52937 

52957.2 

52964. 

52964.5 

.52972. 

52991 

0.102 

2019 

53014 

53039.2 

53050 

53049.7 

53059. 

53096 

0.155 

2020 

53027 

53058.2 

53066 

53069.0 

53080. 

53125 

0.185 

2021 

53101 

53121 

53134 

53136.4 

53150 

53202 

0.19 

Desian Point 2 

2015 

52305 

52310 

52312 

52312.0 

52315 

52322 

0.032 

2016 

52361 

52373 

52377. 

52377.6 

52382 

52394 

0.063 

2017 

52459 

52474 

52480 

52479.9 

52486 

52505 

0.088 

2018 

52534 

52555 

52560 

52560.7 

52567. 

52583 

0.093 

2019 

52515 

52538.2 

52550 

52549.9 

52557. 

52594 

0.15 

2020 

52490 

52530 

52540. 

52540.3 

52552 

52585 

0.181 

2021 

52508 

52544 

52555. 

52556.1 

52570 

52619 

0.211 

Desian Point 3 

2015 

51758 

51761 

51764 

51764.1 

51766. 

51773 

0.029 

2016 

51324 

51337.2 

51341 

51341.7 

51346 

51359 

0.068 

2017 

50981 

50994 

51000 

51000.1 

51006 

51020 

0.076 

2018 

50643 

50666 

50672 

50673.1 

50680 

50700 

0.112 

2019 

50282 

50303 

50314 

50313.9 

50325 

50352 

0.139 

2020 

49955 

49984.2 

49996 

49996.1 

50006 

50049 

0.188 

2021 

49729 

49764 

49772 

49774.1 

49785. 

49841 

0.225 

Desian Point 4 

2015 

52504 

52510 

52512 

52511.7 

52514 

52520 

0.03 

2016 

52752 

52763 

52766 

52766.0 

52769 

52782 

0.057 

2017 

53018 

53032.2 

53038 

53037.8 

53044 

53056 

0.072 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

58214 

53232.2 

53241 

53239.6 

53247 

53259 

0.085 

2019 

53339 

53367 

53376 

53376.0 

53383. 

53410 

0.133 

2020 

53419 

53436 

53445 

53444.4 

53454. 

53469 

0.094 

2021 

53514 

53538 

53551. 

53552.6 

53564. 

53614 

0.187 

Desian Point 5 

2015 

52382 

52389.2 

52392 

52392.0 

52395 

52399 

0.032 

2016 

52513 

52527 

52530 

52530.1 

52534. 

52544 

0.059 

2017 

52673 

52691 

52696 

52696.3 

52703 

52718 

0.085 

2018 

52809 

52824 

52831 

52832.6 

52839. 

52869 

0.114 

2019 

52832 

52861.2 

52871. 

52872.0 

52884 

52929 

0.183 

2020 

52853 

52890.2 

52903 

52904.6 

52920 

52947 

0.178 

2021 

52903 

52944 

52960. 

52959.6 

52979. 

53018 

0.217 

Desian Point 6 

2015 

51834 

51841 

51843 

51843.4 

51846 

51857 

0.044 

2016 

51474 

51488 

51493 

51492.5 

51496. 

51507 

0.064 

2017 

51187 

51203 

51211 

51210.2 

51217 

51235 

0.094 

2018 

50906 

50928 

50934. 

50935.3 

50943. 

50963 

0.112 

2019 

50592 

50608.2 

50618 

50619.7 

50631 

50666 

0.146 

2020 

50299 

50327.2 

50337 

50337.2 

50347. 

50368 

0.137 

2021 

50107 

50140.2 

50150 

50151.7 

50163 

50206 

0.197 

Desian Point 7 

2015 

52624 

52630 

52632 

52632.1 

52634. 

52641 

0.032 

2016 

52980 

52991 

52995 

52994.8 

52999 

53009 

0.055 

2017 

53339 

53356 

53362 

53362.6 

53369. 

53385 

0.086 

2018 

53630 

53643 

53650. 

53652.8 

53662 

53687 

0.106 

2019 

53832 

53854 

53865 

53866.6 

53877. 

53913 

0.15 

2020 

53966 

53990.2 

54001 

54002.1 

54014. 

54042 

0.141 

2021 

54129 

54157.5 

54173. 

54173.2 

54187. 

54225 

0.177 

Desian Point 8 

2015 

52502 

52509 

52511 

52511.5 

52514 

52520 

0.034 

2016 

52746 

52753 

52757 

52757.4 

52761 

52780 

0.064 

2017 

53002 

53014.2 

53019 

53019.9 

53025 

53042 

0.075 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

53219 

53232 

53238 

53239.3 

53246. 

53265 

0.086 

2019 

53316 

53338 

53351 

53351.9 

53365. 

53394 

0.146 

2020 

53404 

53434 

53447 

53449.0 

53466. 

53498 

0.176 

2021 

53512 

53544.2 

53555. 

53557.9 

53570 

53607 

0.177 

Desian Point 9 

2015 

51955 

51961 

51964 

51963.8 

51966 

51973 

0.035 

2016 

51700 

51711 

51715 

51715.3 

51719 

51730 

0.058 

2017 

51509 

51521 

51526 

51526.3 

51531. 

51546 

0.072 

2018 

51306 

51323 

51331 

51330.6 

51336. 

51360 

0.105 

2019 

51051 

51073.2 

51082 

51082.5 

51089. 

51128 

0.151 

2020 

50819 

50849 

50861 

50860.7 

50872 

50906 

0.171 

2021 

50693 

50713.2 

50726 

50726.1 

50737 

50775 

0.162 

Desia 

n Point 10 

2015 

51894 

51903.2 

51907 

51906.8 

51910 

51926 

0.062 

2016 

51548 

51568.2 

51575 

51574.8 

51580 

51601 

0.103 

2017 

51251 

51269.2 

51277 

51277.2 

51284 

51318 

0.131 

2018 

50889 

50926.2 

50936. 

50935.4 

50943. 

50963 

0.145 

2019 

50509 

50540.2 

50552 

50549.9 

50560 

50584 

0.148 

2020 

50106 

50139.2 

50152 

50150.7 

50162 

50189 

0.166 

2021 

49781 

49813.7 

49829 

49828.0 

49841 

49870 

0.179 

Desia 

n Point 11 

2015 

51906 

51917.2 

51921 

51921.5 

51925 

51948 

0.081 

2016 

51601 

51617.2 

51626 

51625.0 

51632 

51654 

0.103 

2017 

51352 

51370 

51378 

51377.0 

51385 

51403 

0.099 

2018 

51080 

51106 

51114. 

51115.7 

51124. 

51161 

0.158 

2019 

50752 

50776 

50787. 

50788.1 

50800 

50833 

0.159 

2020 

50457 

50484.2 

50502. 

50498.9 

50513 

50539 

0.162 

2021 

50199 

50240.2 

50256 

50255 

50269. 

50308 

0.217 

Desia 

n Point 12 

2015 

51685 

51699 

51704 

51703.8 

51709 

51716 

0.06 

2016 

51193 

51211 

51224 

51221.2 

51230. 

51244 

0.1 

2017 

50796 

50819 

50829 

50828.0 

50837 

50863 

0.132 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

50406 

50423.2 

50435 

50434.7 

50444 

50467 

0.121 

2019 

49953 

49993 

50003. 

50004.5 

50014. 

50068 

0.23 

2020 

49573 

49599 

49617 

49615.5 

49628. 

49688 

0.232 

2021 

49281 

49313 

49331 

49329.4 

49345 

49382 

0.205 

Desia 

n Point 13 

2015 

51971 

51983 

51986. 

51986.1 

51990 

52000 

0.056 

2016 

51699 

51720.2 

51727 

51726.5 

51732. 

51747 

0.093 

2017 

51467 

51486.2 

51495 

51494.7 

51503. 

51525 

0.113 

2018 

51170 

51202 

51212 

51210.6 

51221 

51239 

0.135 

2019 

50839 

50864 

50876 

50875.7 

50887. 

50917 

0.153 

2020 

50491 

50512 

50525. 

50525.4 

50536 

50584 

0.184 

2021 

50196 

50231 

50247. 

50248.4 

50265 

50325 

0.257 

Desia 

n Point 14 

2015 

51984 

51998 

52002. 

52002.4 

52008 

52018 

0.065 

2016 

51752 

51772 

51781 

51778.9 

51786. 

51806 

0.104 

2017 

51563 

51585.2 

51596 

51595.0 

51604 

51624 

0.118 

2018 

51347 

51379.2 

51390 

51388.8 

51401. 

51426 

0.154 

2019 

51067 

51096.2 

51110. 

51109.7 

51124 

51151 

0.164 

2020 

50801 

50844.2 

50862. 

50864.1 

50881. 

50917 

0.228 

2021 

50601 

50641.2 

50658. 

50660.3 

50680 

50727 

0.249 

Desia 

n Point 15 

2015 

51769 

51780 

51785 

51784.4 

51788 

51800 

0.06 

2016 

51348 

51366.2 

51372. 

51372.6 

51379 

51395 

0.091 

2017 

51011 

51031.2 

51040 

51040.9 

51048 

51088 

0.151 

2018 

50667 

50689.2 

50696. 

50699.0 

50710 

50757 

0.178 

2019 

50272 

50300 

50312 

50312.6 

50325 

50361 

0.177 

2020 

49931 

49949.2 

49961 

49963.2 

49975 

50005 

0.148 

2021 

49660 

49698 

49713 

49710.8 

49723 

49758 

0.197 

Desia 

n Point 16 

2015 

52096 

52102.2 

52106 

52105.9 

52109 

52118 

0.042 

2016 

51940 

51949.2 

51955 

51955.5 

51961 

51981 

0.079 

2017 

51790 

51810 

51820 

51819.5 

51827. 

51851_ 

0.118 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

51587 

51611 

51621 

51621.0 

51631 

51654 

0.13 

2019 

51328 

51350 

51361 

51361.8 

51374. 

51401 

0.142 

2020 

51044 

51065.7 

51080. 

51079.9 

51093. 

51118 

0.145 

2021 

50815 

50850.2 

50867 

50866.8 

50880. 

50936 

0.238 

Desia 

n Point 17 

2015 

52104 

52117 

52123 

52121.9 

52126 

52134 

0.058 

2016 

51977 

51997 

52004 

52004.2 

52013 

52028 

0.098 

2017 

51886 

51909.5 

51917. 

51916.8 

51927 

51947 

0.117 

2018 

51756 

51782.2 

51792 

51794.3 

51807 

51837 

0.156 

2019 

51546 

51573 

51591 

51590.3 

51604 

51646 

0.194 

2020 

51351 

51394.2 

51408 

51407.5 

51421. 

51451 

0.195 

2021 

51202 

51250.2 

51266. 

51264.3 

51281. 

51328 

0.246 

Desia 

n Point 18 

2015 

51884 

51899 

51905 

51903.9 

51908. 

51923 

0.075 

2016 

51568 

51588 

51597 

51596.1 

51603 

51628 

0.116 

2017 

51326 

51347 

51355. 

51356.0 

51365 

51385 

0.115 

2018 

51060 

51081 

51091 

51093.1 

51107 

51137 

0.151 

2019 

50725 

50758 

50771. 

50773.2 

50785 

50833 

0.213 

2020 

50414 

50463.2 

50482 

50481.5 

50498 

50541 

0.252 

2021 

50225 

50269.2 

50284. 

50284.4 

50302 

50350 

0.249 

Desia 

n Point 19 

2015 

51349 

51352 

51353 

51353.2 

51354 

51358 

0.018 

2016 

53128 

53131 

53132 

53131.7 

53133 

53135 

0.013 

2017 

50337 

50340 

50341 

50340.9 

50342 

50344 

0.014 

2018 

50679 

50681 

50682 

50682.1 

50683 

50685 

0.012 

2019 

50733 

50736 

50736 

50736.4 

50737 

50741 

0.016 

2020 

50736 

50740 

50741 

50740.6 

50742 

50744 

0.016 

2021 

50737 

50740 

50741 

50740.8 

50742 

50745 

0.016 

Desia 

n Point 20 

2015 

51348 

51352 

51353 

51353.2 

51354 

51358 

0.019 

2016 

53126 

53128 

53129 

53129.3 

53130 

53133 

0.013 

2017 

50338 

50341 

50342 

50342.3 

50344 

50347 

0.018 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

50678 

50680 

50681 

50680.7 

50682 

50684 

0.012 

2019 

50733 

50737 

50738 

50738.2 

50740 

50742 

0.018 

2020 

50739 

50741 

50742 

50741.7 

50743 

50744 

0.01 

2021 

50738 

50741 

50741. 

50741.6 

50743 

50746 

0.016 

Desia 

n Point 21 

2015 

51350 

51352 

51353 

51353.3 

51354 

51357 

0.014 

2016 

53125 

53129 

53130 

53130.0 

53131 

53133 

0.015 

2017 

50337 

50339 

50340 

50340.0 

50341 

50343 

0.012 

2018 

50678 

50681 

50683 

50682.4 

50684 

50686 

0.016 

2019 

50734 

50737 

50738 

50738.2 

50739. 

50742 

0.016 

2020 

50735 

50738 

50739 

50739.6 

50741 

50745 

0.02 

2021 

50737 

50739 

50740 

50740.3 

50742 

50746 

0.018 

Desia 

n Point 22 

2015 

51349 

51351 

51353 

51352.5 

51354 

51356 

0.014 

2016 

53129 

53131 

53132 

53132.0 

53133 

53136 

0.013 

2017 

50336 

50339 

50340 

50339.5 

50340 

50345 

0.018 

2018 

50675 

50679 

50681 

50680.8 

50682 

50686 

0.022 

2019 

50736 

50738 

50739 

50738.8 

50740 

50743 

0.014 

2020 

50737 

50740 

50740 

50740.3 

50741 

50743 

0.012 

2021 

50738 

50740 

50741 

50740.7 

50742 

50744 

0.012 

Desia 

n Point 22 

2015 

51350 

51352 

51352 

51352.5 

51354 

51356 

0.012 

2016 

53126 

53129 

53131 

53130.8 

53132 

53137 

0.021 

2017 

50338 

50341 

50342 

50341.7 

50343 

50346 

0.016 

2018 

50678 

50680 

50681 

50680.9 

50682 

50684 

0.012 

2019 

50735 

50737 

50738 

50738.3 

50739. 

50743 

0.016 

2020 

50737 

50739 

50740. 

50740.5 

50742 

50744 

0.014 

2021 

50736 

50738 

50739 

50739.5 

50741 

50743 

0.014 

Desia 

n Point 24 

2015 

51348 

51351 

51352 

51352.2 

51353 

51356 

0.016 

2016 

53128 

53131 

53131. 

53131.4 

53132 

53135 

0.013 

2017 

50337 

50340 

50341 

50340.8 

50342 

50344 

0.014 
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Fiscal 

Year 

Minimum 

25% 

Quartile 

Median 

Mean 

75% 

Quartile 

Maximum 

Range 

(as % 

Mean) 

2018 

50678 

50680 

50681 

50680.7 

50682 

50684 

0.012 

2019 

50733 

50735 

50737 

50736.6 

50738 

50741 

0.016 

2020 

50737 

50740 

50741 

50741.2 

50742 

50745 

0.016 

2021 

50738 

50740 

50742 

50741.5 

50743 

50746 

0.016 

Desia 

n Point 25 

2015 

51350 

51353 

51354 

51354.1 

51355 

51357 

0.014 

2016 

53129 

53132 

53133 

53132.6 

53134 

53136 

0.013 

2017 

50339 

50341 

50343 

50342.4 

50344 

50346 

0.014 

2018 

50676 

50679 

50680 

50680.3 

50682 

50684 

0.016 

2019 

50735 

50738 

50739 

50738.9 

50740 

50742 

0.014 

2020 

50737 

50740 

50741 

50740.6 

50742 

50744 

0.014 

2021 

50734 

50739 

50740 

50740 

50741 

50744 

0.02 

Desia 

n Point 26 

2015 

51349 

51351 

51352 

51351.9 

51353 

51355 

0.012 

2016 

53126 

53133 

53134 

53134.2 

53135 

53138 

0.023 

2017 

50338 

50339.2 

50340 

50340.4 

50341 

50344 

0.012 

2018 

50679 

50681 

50682 

50682.2 

50683 

50686 

0.014 

2019 

50735 

50738 

50739 

50739.3 

50740 

50743 

0.016 

2020 

50735 

50737 

50738 

50737.8 

50739 

50742 

0.014 

2021 

50734 

50737 

50738 

50738.3 

50739 

50744 

0.02 

Desia 

n Point 27 

2015 

51348 

51351 

51352 

51351.9 

51353 

51356 

0.016 

2016 

53129 

53131.2 

53132 

53132.3 

53133 

53135 

0.011 

2017 

50340 

50344 

50344. 

50344.5 

50345. 

50348 

0.016 

2018 

50677 

50680 

50681 

50681.3 

50682 

50685 

0.016 

2019 

50736 

50738 

50739 

50739.0 

50740 

50742 

0.012 

2020 

50738 

50740 

50741 

50741.2 

50742 

50744 

0.012 

2021 

50738 

50739 

50740 

50740.3 

50741 

50744 

0.012 
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APPENDIX C. ANOVA AND TUKEY-KRAMER HSD 

TEST RESULTS 


Oneway Analysis of Standard Deviation(End Strength) By Design Point 
▼ Oneway Anova 


Analysis of Variance 


Source 

DF 

Sum of 
Squares 

Mean Square F Ratio Prob > F 

Decision Point 

26 

6467.144 

248.736 10.0430 <.0001* 

Error 

162 

4012.268 

24.767 

C. Total 

188 

10479.412 



*■ Means for Oneway Anova 


▼ Means Comparisons 

▼ " Comparisons for all pairs using Tukey-Kramer HSD 
*■ Confidence Quantile 
*■ HSD Threshold Matrix 
Connecting Letters Report 


Level 


Mean 

9 

A B 

11.092679 

8 

A 

12.003696 

7 

A 

12.116725 

6 

A B 

11.057775 

5 

A 

13.346556 

4 

A B 

10.049009 

3 

A B 

11.493515 

27 

B 

1.510071 

26 

B 

1.543125 

25 

B 

1.574443 

24 

B 

1.621270 

23 

B 

1.601490 

22 

B 

1.505326 

21 

B 

1.707909 

20 

B 

1.599568 

2 

A 

11.984277 

19 

B 

1.508218 

18 

A 

16.679245 

17 

A 

16.720279 

16 

A 

14.127116 

15 

A 

14.118529 

14 

A 

17.029345 

13 

A 

14.344529 

12 

A 

15.667221 

11 

A 

14.599929 

10 

A 

13.611234 

1 

A 

12.273249 


" Oneway Analysis of Standard Deviation(End 
Strength) By Accession Option 
» Oneway Anova 
▼ Analysis of Variance 


Sum of 


Source 

DF 

Squares 

Mean Square 

F Ratio 

Prob>F 

Accession Option 

2 

6320.353 

3160.18 

141.3283 

<.0001* 

Error 

186 

4159.059 

22.36 



C. Total 

188 

10479.412 





*■ Means for Oneway Anova 


▼ Means Comparisons 

^ Comparisons for all pairs using Tukey-Kramer HSD 
» Confidence Quantile 
* HSD Threshold Matrix 
» Connecting Letters Report 

Level Mean 

1 A 15.210825 

0 B 11.713053 

2 C 1.574602 
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Oneway Analysis of Standard Deviation(End Strength) By Loss Plan 
▼ Oneway Anova 

Analysis of Variance 


Sum of 


Source 

DF 

Squares 

Mean Square 

F Ratio 

Prob>F 

Loss Plan 

2 

34.118 

17.0589 

0.3038 

0.7384 

Error 

186 

10445.294 

56.1575 



C. Total 

188 

10479.412 





Means for Oneway Anova 
▼ Means Comparisons 

* Comparisons for all pairs using Tukey-Kramer HSD 
^ Confidence Quantile 
* HSD Threshold Matrix 
'' Connecting Letters Report 

Level Mean 

0 A 10.047585 

1 A 9.438690 

-1 A 9.012205 


'' Oneway Analysis of Standard Deviation(End Strength) By POOR 
V Oneway Anova 
▼ Summary of Fit 

V Analysis of Variance 


Sum of 


Source 

DF 

Squares 

Mean Square 

F Ratio 

Prob> F 

POOR 

2 

4.109 

2.0543 

0.0365 

0.9642 

Error 

186 

10475.303 

56.3188 



C. Total 

188 

10479.412 





>■ Means for Oneway Anova 
Means Comparisons 

▼ Comparisons for all pairs using Tukey-Kramer HSD 

* Confidence Quantile 

* HSD Threshold Matrix 
Connecting Letters Report 


Level 


Mean 

0 

A 

9.7074865 

120 

A 

9.4082032 

200 

A 

9.3827912 


Oneway Analysis of Standard Deviation(End Strength) By Fiscal Year 
■» Oneway Anova 
► Summary of Fit 
▼ Analysis of Variance 


Sum of 


Source 

DF 

Squares 

Mean Square 

F Ratio 

Prob>F 

Rscal Year 

6 

2577.032 

429.505 

9.8920 

<.0001* 

Error 

182 

7902.380 

43.420 



C. Total 

188 

10479.412 





> Means for Oneway Anova 
▼ Means Comparisons 

▼ Comparisons for all pairs using Tukey-Kramer HSD 
> Confidence Quantile 
>• HSD Threshold Matrix 
▼ Connecting Letters Report 

Level Mean 

2021 A 14.900347 

2020 A B 12.985708 

2019 A B 11.847254 

2018 B C 9.271352 
2017 BCD 7.733471 
2016 C D 6.050676 
2015 D 3.707646 
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